Trace of a bd-management run showed the keystone CONFIRMING a true-but-wrong cause:
'bd-management is not deployed in the cluster (no k8s pod exists)'. It's true (a
Consul service has no k8s pod) but it is NOT the root cause — and the run had
already gathered consul_health evidence proving the service is bare-metal Consul.
The skill tells the model not to do this, but it does ~half the time; a prompt
can't guarantee it.

Deterministic guard in the conclude path: if the run gathered any
consul_health_service_status evidence AND the confirmed cause asserts a missing
k8s deployment ('not deployed', 'no k8s pod', 'deployment missing/absent'), reject
it and make the agent confirm via the Consul health signal instead. Genuine k8s
incidents (e.g. agw-admin-ui's deleted namespace) gather no consul_health
evidence, so the guard never fires for them — proven by a control test.

tsc clean; 2 new tests (guard fires for Consul / allows genuine k8s); suite green.

Validation surfaced a second wrong-layer failure: investigating the idle
minimax-m25-metrics-proxy, the orchestrator ruled out every real hypothesis, then
chased its OWN query errors — 'datasource with UID loki not found' (MI350 has no
Loki datasource configured) appearing as 404s in logs — and confirmed 'Grafana
datasource misconfigured' as the root cause. That's a query-layer artifact, not
why the service is unhealthy.

Guard (same pattern as the Consul category-error guard, same conclude path): reject
a confirmed cause that blames the observability tooling (mentions 'datasource' or
'grafana') UNLESS the incident service is itself an observability component
(grafana/loki/prometheus/...). Forces the agent back to the service's own signals;
an idle service with no real cause then exhausts/pauses honestly instead of
inventing a tooling cause.

tsc clean; 2 new tests (guard fires for a service / allows a grafana incident); suite green.

Investigating minimax-m25-vllm-bench-4gpu (inv_01KTS9EK21G042S6YAS5MSCTE7), every
log query came back empty and the orchestrator kept reporting 'couldn't verify (no
evidence gathered)'. Root cause: gpt-oss generates an OR-chain of repeated line
filters to search multiple terms —

  {container_name="svc"} |= "replicas" or {container_name="svc"} |= "scale" or {container_name="svc"} |= "deployment"

which is INVALID LogQL (`or` joins label-filter expressions, not `{sel} |=`
line-filter pipelines). Loki rejects it with HTTP 400 ('parse error ... unexpected
{'), so the gather returns ZERO log observations → no log evidence for any
log-based hypothesis (and contributes to MI350 investigations going inconclusive).

Fix: coerceLokiArgs (already on the Loki tool exec path, tool-utils.ts:345) now
detects the repeated-selector OR-chain and collapses it into the correct single
regex line filter — {sel} |~ "replicas|scale|deployment". Single filters and
already-regex queries pass through untouched; terms are deduped + regex-escaped.

tsc clean; 3 new tests; full suite green.
Symptom: log queries 400 → 'no evidence gathered'. Root cause: invalid OR-chain
LogQL. Fix: tool-utils.ts coerceLokiArgs. Regression: tool-utils.test.ts:84.

…-pause

Investigating minimax-m25-vllm-bench-4gpu (a k8s Deployment scaled to 0 replicas,
kube_deployment_spec_replicas=0), the orchestrator spent all ~30 moves chasing
Consul hypotheses ('is it a bare-metal Consul service / not registered in Consul /
health check unknown...'), never proposed the obvious 'deployment scaled to 0',
and operator-paused. Root cause: #256 injects the Consul runbook into EVERY
orchestrator run (always-on), which over-steers k8s services into Consul
tunnel-vision — a regression for the k8s case introduced by the Consul fixes.

Two parts:
- Skill: lead the 'When investigating' section with DETERMINE SERVICE TYPE FIRST —
  query kube_deployment_status_replicas; if it returns data the service IS a k8s
  Deployment (0 = scaled to 0 = a valid root cause, conclude it), and ONLY treat as
  Consul when there's no kube_deployment metric. Stops the Consul tunnel-vision.
- Guard: the Consul category-error guard no longer fires when the run also saw
  kube_deployment/kube_pod evidence — a real k8s Deployment's k8s-absence cause is
  legitimate (only consul_health-AND-no-kube services are the bare-metal ones).

tsc clean; 1 new test (k8s service not blocked); suite green.

…oki-accuracy

…racy

…skill

The orchestrator and ws-handler had hardcoded Consul-specific logic (a
conclude-time category-error guard keyed on consul_health_service_status,
and a skill-injection scope that special-cased the Consul runbook). That
coupled the engine to one infrastructure type.

Replace it with a generic mechanism: a skill declares an optional
`appliesToServiceMetric` substring in its frontmatter, and the engine
injects that skill only when the incident service's discovered metric
queries contain it. The Consul runbook now declares
`appliesToServiceMetric: consul_health_service_status` — all infra-type
knowledge lives in the skill, none in the engine.

- orchestrator.ts: drop the Consul category-error guard (keep the generic
  observability-artifact guard)
- ws-handler.ts: generic appliesToServiceMetric filter, no consul_health regex
- hypothesis-requery.ts: generalize the Consul aggregation hint to any bare
  (un-aggregated) metric selector
- store.ts: parse + round-trip the appliesToServiceMetric field

A k8s service (e.g. a Deployment scaled to 0) now receives zero Consul
context and diagnoses cleanly; Consul services still get the runbook via
the declared metric match.

The keystone could not confirm an absence. When the brain proposed the
correct cause for a workload scaled to zero replicas, the prediction's
gather returned no data (pod-runtime metrics vanish at zero replicas), so
evaluatePrediction scored it 'absent' → 'couldn't verify', the correct
cause was discarded, and the run drifted to a wrong cause that happened to
gather evidence (GPU OOM, cross-service blame) or wall-clocked. Across
repeated runs on the same scaled-to-zero incident, only the run that
happened to phrase a luckily-verifiable prediction succeeded.

Two generic, infra-agnostic verification paths for an absence:
- (B) state metric that survives at zero: the decide-move prompt steers the
  brain to predict over a STATE metric (e.g. kube_deployment_status_replicas
  < 1) that still reports 0, instead of a runtime metric that disappears.
- (A) explicit absence prediction: extend the existing log-pattern
  present:false mechanism to infra-status. evaluatePrediction confirms an
  absence only when infra evidence was actually gathered (mirrors the
  log-pattern haveLogs guard — never confirm on a query that may not have run).

schemas.ts threads present through the decide-move parse; hypothesis-requery
asks the gather to report the resource's actual state including its absence.

The absence-verification fix made 'scaled to zero' confirmable — which is
correct for a genuine k8s workload but ALSO let a Consul service be falsely
confirmed as 'a Kubernetes Deployment scaled to zero' (5x re-run: impala,
a consul_health-tracked service, hit this 1/5). With the hardcoded Consul
guard removed, nothing enforced that a conclusion's infra framing matches
the service's actual type.

Add a generic conclude-time guard: derive the service's infrastructure type
from its discovered identity metric FAMILY (kube_* → k8s, consul_* → Consul)
and reject a confirm whose framing contradicts it — a k8s/deployment/replica
cause for a Consul service, or a Consul cause for a kube_deployment workload.
Driven by the metric family, not a per-service or Consul-only list; silent
when the identity is an unrecognised family (up{}, etc.).

Threads the incident service's identity metrics (ws-handler → agents.orchestrate
→ runAutonomousOrchestrator → OrchestratorDeps). Also catches the reverse
hallucination seen in the re-run (a k8s service confirmed with a 'consul
registration' tail).

…blind)

The orchestrator is LLM-driven and non-deterministic — the same incident can
confirm the right cause, wall-clock, or confirm a wrong one across runs. Until
now 'quality is bad' was a vibe from slow manual batches; there was no way to
tell whether a change helped.

This harness turns a batch of orchestrator runs into objective rates:
- src/eval/deep-eval.ts — pure scorer + CLI. Scores each run against a label:
  correct / confident-wrong / category-error / honest-inconclusive. Headline
  metrics: correctRate and confidentWrongRate (the 'do no harm' bar, target 0).
  --save / --compare / --max-confident-wrong / --max-category-error / --min-correct.
- src/eval/deep-investigation-run.mjs — live batch runner (baseline → orchestrator,
  auto-continue at operator-pauses), writes a results JSON the scorer reads. Makes
  the previously-manual batch reproducible.
- fixtures/deep-investigation-labels.json — labeled incidents (infra type, expected
  cause keywords, wrong-cause patterns, acceptable inconclusive outcomes).
- baselines/deep-2026-06-11.json — first scorecard after Fix 1:
  40% correct, 20% confident-wrong (the GPU fabrication), 0 category-error.

10 scorer unit tests. Establishes the measurement loop to gate the un-gate on
0% category-error / 0% fabrication / correct-on-clear-cases.

Two measured-against-baseline accuracy fixes, both driven by the same
incidentServiceMetrics identity signal Fix 1 introduced.

Fix 2 — identity-steering (Consul conclusiveness). The brain often never
proposed the hypothesis that works: for a Consul service it wandered into k8s
framings or dependency-chasing instead of testing consul_health, and
wall-clocked (impala 2/3). When it DID propose 'Consul health failing' it
verified fine (+4 obs). serviceIdentityHint derives a one-line steer from the
service's identity metric family and prepends it to the decide-move prompt:
Consul service → test consul_health_service_status < 1 first; k8s workload →
establish replica state first. Positive mirror of the Fix 1 guard.

Fix 3 — grounding gate (kill the fabrication). The 20% confident-wrong in the
baseline was minimax confirming 'GPU resource exhaustion' for a deployment
scaled to zero — there are no pods at 0 replicas, so no GPU/OOM/readiness
failure is possible (cluster-wide GPU usage misattributed). Conclude-time
guard: if gathered evidence shows zero running pods, reject a confirm that
blames a pod-runtime failure (GPU/CUDA/OOM/readiness/crash-loop/CPU). Only
fires when an observation actually establishes zero replicas, so it's grounded.

+6 unit tests; 2731 green. Re-score against baselines/deep-2026-06-11.json next.

…op stale baselines

The MI350 stack was removed, so the minimax (k8s scaled-to-zero) label is
retired. Re-key the eval labels to services present on the current bigdata
cluster stacks: Consul services (impala, kudu, hdfs-namenode,
bd-management-task-management) and k8s workloads (ingestion-server,
bd-management-task). The k8s 'bd-management-task' vs Consul
'bd-management-task-management' pair is a direct category-error probe.

Delete the MI350-era baselines (deep-2026-06-11.json, deep-latest.json) so the
next run starts fresh; --save regenerates the baseline. CLAUDE.md updated to
the --save / CI-gate command pair.

…fident-wrong)

8-incident run across the two live stacks. Corrected the labels: a Consul
service can be Consul-healthy while failing on its own daemon or a real
dependency, so a grounded on-type cause is CORRECT — only a cross-type
category error or a fabrication is wrong. Added timeout-orch as an honest
non-confirm.

Result (deep-2026-06-15): 0% category-error, 0% confident-wrong, 0
fabrications. 3 grounded confirms (impala daemon, ingestion-server CPU,
Cortex ingester disk), 4 honest declines, 1 excluded (network baseline-fail).
Both Consul confirms tried consul_health FIRST then ruled it out when healthy
(Fix 2 working). Meets the un-gate safety bar on real cluster traffic.

The single 'Consul Bare-Metal Service Discovery' skill was doing double duty
(scope: [discovery, investigation]) — its title says discovery but it also
carried the RCA/investigation runbook. Split into two single-purpose skills so
Consul knowledge stays in skills (never the engine) and each is scoped right:

- consul-bare-metal-discovery (scope: discovery) — finding Consul services +
  writing registry entries, for the discovery agent. No appliesToServiceMetric.
- consul-bare-metal-investigation (scope: investigation, appliesToServiceMetric:
  consul_health_service_status) — RCA runbook, injected into the orchestrator
  ONLY for Consul-tracked services when enabled. Carries the verifiable confirm
  prediction (aggregated passing form, so the gather reduces to a value instead
  of 'couldn't verify') AND an explicit 'healthy != a root cause' rule: never
  confirm 'Consul failing' when the passing row reads 1 — decline instead.

This is the skill-side home for the consul-health verification (no engine
hardcoding) and addresses the ground-truth miss + the over-confirm on healthy
Consul services. 2731 tests green.

…split)

P1b: the k8s-side 'healthy != a root cause' runbook, parallel to the Consul
investigation skill. investigation-scoped, appliesToServiceMetric: kube_ — so
it's injected into the orchestrator only for k8s-tracked services. Carries the
rule that fixes the ingestion-server false-confirm: never confirm CPU-throttling
/ OOM / readiness / crash causes unless the actual throttle/restart/OOM metric
was gathered and crossed a threshold; if replicas available == spec and nothing
abnormal was seen, decline (inconclusive) rather than manufacture a cause.

Keeps infra knowledge in skills, engine generic. 2731 tests green.

…mpt assembly

No prior visibility into whether investigation skills actually reach the
decide-move prompt. Add LOG_LEVEL=debug logs at both points: ws-handler logs
the matched skills (+ appliesToServiceMetric) and skillContext size for the
incident service; orchestrator-llm logs whether skillContext/identityHint are
present and the injected skill titles. Used to prove the Consul investigation
skill IS injected for impala yet the LLM over-confirms anyway — i.e. skill
prompts are followed unreliably, so a deterministic engine gate is needed.

…skills

The decisive finding (debug-log proof): the investigation skill IS injected
into the decide-move prompt, but the LLM routes around its 'healthy != cause'
rule and over-confirms on healthy services. So prompt guidance alone can't
hold — a deterministic engine gate is required — AND the engine still hardcoded
Consul/k8s literals (one always-on in the SYSTEM_PROMPT, distracting non-Consul
stacks).

Fix both: move ALL infra-type knowledge into skill-declared metadata and have
the engine read it generically.

- store: skills declare healthySignal / identityHint / incompatibleClaims
  ($service substituted), parsed + round-tripped like appliesToServiceMetric.
- consul + k8s investigation skills declare those fields.
- orchestrator-llm: drop serviceIdentityHint (the consul/k8s detector); take
  identityHint from the matched skill; generalize the SYSTEM_PROMPT to
  'don't assume a platform; use the injected skills' (removes the always-on
  Consul/k8s/vllm literals → no distraction on a non-Consul stack).
- orchestrator: service-type guard now rejects on the matched skill's
  incompatibleClaims regex (no consul/kube literals); NEW health-gate evaluates
  the matched skill's healthySignal before any confirm — healthy => inconclusive
  (the deterministic backstop the proof demands).
- agents.ts derives identityHint/incompatibleClaims/checkHealthy from the
  matched skills (+ providers for the healthySignal query); prometheus-query
  gains queryInstantValue.

Result: orchestrator engine carries zero Consul literals (only comments + the
eval term-list). A stack with no Consul skill enabled sees zero Consul text.
2732 tests green. (Residual: the k8s grounding-gate still has kube_ literals —
follow-up.)

… filter

Root cause of the ingestion-server over-confirm that survived the refactor:
getAllForScope sliced to maxPerQuery (default 3) by TITLE before the
ws-handler's appliesToServiceMetric filter ran. With 4 investigation skills
(Consul…, Investigate High-CPU, Investigate Kafka, Kubernetes Workload…),
'Kubernetes Workload Investigation' sorts last and was dropped by the cap — so
the k8s skill (and its healthySignal) never reached a k8s service, the
health-gate never ran, and ingestion kept over-confirming.

Fix: getAllForScope returns ALL scoped skills uncapped; ws-handler relevance-
filters by appliesToServiceMetric, orders service-targeted skills FIRST, then
caps to maxPerQuery — so a matched infra skill can never be dropped. Found via
the skill-injection debug log. 2732 tests green.

… real-incident catch)

The remaining gap after the infra-agnostic refactor: bd-management (consul
CRITICAL) confirmed correctly one run and wall-clocked the next — LLM variance
meant a genuinely-broken service wasn't reliably caught.

Fix it deterministically + skill-driven. Skills declare a failureSignal (PromQL
>=1 when the service is definitely failing on its primary signal — consul
status=critical, replicas unavailable) and a failureCause (the grounded cause
text). The engine evaluates failureSignal at every give-up point (wall-clock /
exhausted / tool-cap / budget / inconclusive / operator-pause) via an async
giveUp() wrapper; if it fires, the run confirms the skill's failureCause instead
of missing the incident. Confirms and aborts are untouched; an explicit agent
confirm is never overridden. No infra literals in the engine — failureSignal +
failureCause live in the skills.

+3 unit tests; 2735 green. Verify live next.

F1 — a generic SYSTEM_PROMPT rule: a rate/ratio/percentage/throughput cause
('0% success rate', 'high error rate', 'queries returning zero results') is only
valid with evidence of NON-ZERO traffic; zero requests = idle, not broken. No
infra literals. Complements the health-gate (which already blocks confirms on a
healthy service) for the general idle-misread.

F2 — ground-truth-anchored scoring, the metric that actually tracks quality
(keyword scoring gave a false PASS — it can't tell a grounded-sounding
false-confirm from a real cause). deep-eval.ts gains scoreRunsGroundTruth +
--ground-truth: score each run against the service's REAL health —
confirm-on-healthy = false-confirm, decline-on-broken = missed-incident (both
target 0), with --max-false-confirm / --max-missed gates. deep-investigation-
groundtruth.mts fetches the truth via grafana-mcp using the skills' own
healthySignal/failureSignal (consistent with the engine). +6 scorer tests,
including an aggregate that reproduces the real 8-run finding (2 false-confirms
+ 1 missed) — i.e. the harness would now catch what the keyword scorer missed.